Identification using Audio Signatures and Additional Characteristics

ABSTRACT

Techniques for using both speaker-identification information and other characteristics associated with received voice commands to determine how and whether to respond to the received voice commands. A user may interact with a device through speech by providing voice commands. After beginning an interaction with the user, the device may detect subsequent speech, which may originate from the user, from another user, or from another source. The device may then use speaker-identification information and other characteristics associated with the speech to attempt to determine whether or not the user interacting with the device uttered the speech. The device may then interpret the speech as a valid voice command and may perform a corresponding operation in response to determining that the user did indeed utter the speech. If the device determines that the user did not utter the speech, however, then the device may refrain from taking action on the speech.

BACKGROUND

Homes are becoming more wired and connected with the proliferation ofcomputing devices such as desktops, tablets, entertainment systems, andportable communication devices. As computing devices evolve, manydifferent ways have been introduced to allow users to interact withthese devices, such as through mechanical means (e.g., keyboards, mice,etc.), touch screens, motion, and gesture. Another way to interact withcomputing devices is through speech.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 shows an illustrative voice interaction computing architectureset in a home environment. The architecture includes a voice-controlleddevice physically situated in the home, along with a user who isuttering a command to the voice-controlled device.

FIG. 2 depicts a flow diagram of an example process for performing afirst operation in response to receiving a voice command from a user,receiving a second voice command, and performing a second operation inresponse to determining that the user that uttered the first voicecommand also uttered the second voice command.

FIG. 3 depicts a flow diagram of an example process for receiving speechwhile performing an operation for a user and determining whether toperform another operation specified by the speech based on a confidencelevel regarding whether or not the user uttered the speech.

FIG. 4 depicts a flow diagram of an example process for determiningwhether or not to interpret speech as a valid voice command based onwhether a user that utters the speech is the same as a user that uttereda prior voice command.

FIG. 5 depicts a flow diagram of an example process for receiving afirst voice command, performing a first action in response, anddetermining whether or not to perform a second action associated with asecond voice command based on a characteristic of the second actionand/or the second voice command.

FIG. 6 shows a block diagram of selected functional componentsimplemented in the voice-controlled device of FIG. 1.

DETAILED DESCRIPTION

This disclosure describes, in part, techniques for using bothspeaker-identification information and other characteristics associatedwith received voice commands to determine how and whether to respond tothe received voice commands. As described below, a user may interactwith a device through speech by providing one or more voice commands.After beginning an interaction with the user, the device may detectsubsequent speech, which may originate from the user, from another user,or from another source (e.g., a television in the background, a radio,etc.). The device may then use speaker-identification information andother characteristics associated with the speech to attempt to determinewhether or not the user interacting with the device uttered the speech.The device may then interpret the speech as a valid voice command andmay perform a corresponding operation in response to determining thatthe user did indeed utter the speech. If the device determines that theuser did not utter the speech, however, then the device may refrain fromtaking action on the speech. In some instances, however, the devicedetermines whether the user that uttered the subsequent speech isauthorized to instruct the device to perform an action. For instance,envision that the father of a family issues a first voice command andthat the device identifies the father as issuing this command. Thedevice may subsequently allow the mother of the family to interact withthe device regarding the first command (e.g., pausing music that thefather started), while not allowing children in the family to do so.

To provide an example, envision that a first user interacts with acomputing device through speech by, for example, providing a voicecommand requesting that the device play a particular song, make a phonecall for the user, purchase an item on behalf of the user, add areminder to a reminder list, or the like. In response, the device mayperform the corresponding operation for the user. For example, the firstuser may request, via a voice command, to begin playing music on thedevice or on another device. After the device begins playing the music,the first user may continue to provide voice commands to the device,such as “stop”, “next song”, “please turn up the volume”, and the like.

In response to receiving speech and identifying a potential voicecommand, however, the device may first ensure that the command is valid.In one example, the device may first ensure that the first user, whoinitially interacted with the device, is the user providing the voicecommand. If so, then the device may comply with the command. If not,then the device may refrain from complying with the command, which mayinclude querying the first user or another user to ensure the user'sintent and/or to receive authorization to perform the operation from thefirst user. In another example, the device may determine whether theuser that issued the subsequent command is one of a group of one or moreusers that are authorized to do so.

In the instant example, envision that first user and the device residewithin an environment that includes two other users. Furthermore, afterthe device complies with the first user's command and begins playingmusic within the environment, the device may identify speech from one orall of the three users. For instance, envision that the device generatesan audio signal that includes the second user telling the third user to“remember to stop by the grocery store”. The device, or another device,may identify the word “stop” from the audio signal, which if interpretedas a valid voice command may result in the device stopping the playingof the music on the device. Before doing so, however, the device may useboth speaker identification and other characteristics to determinewhether to respond to the command.

In some instances, the device, or another device, may determine whetherthe first user issued the command prior to stopping the music inresponse to identifying the word “stop” from the generated audio signal.To do so, the device may compare a voice signature associated with thefirst user to a voice signature associated with the received speech(“remember to stop by the grocery store”). A voice signature mayuniquely represent a user's voice and may be based on a combination ofone or more of a volume (e.g., amplitude, decibels, etc.), pitch, tone,frequency, and the like. Therefore, the device(s) may compare a voicesignature of the first user (e.g., computed from the initial voicecommand to play the music) to a voice signature associated with thereceived speech. The device(s) may then calculate a similarity of thevoice signatures to one another.

In addition, the device(s) may utilize one or more characteristics otherthan voice signatures to determine whether or not the first userprovided the speech and, hence, whether or not to interpret the speechas a valid voice command. For instance, the device may utilize asequence or choice of words, grammar, time of day, a location within theenvironment from which speech is uttered, and/or other contextinformation to determine whether the first user uttered the speech “stop. . . ” In the instant example, the device(s) may determine, from thespeaker-identification information and the additional characteristics,that the first user did not utter the word “stop” and, hence, mayrefrain from stopping playback of the audio. In addition, the devicewithin the environment may query the first user to ensure the device hasmade the proper determination. For instance, the device may output thefollowing query: “Did you say that you would like to stop the music?” Inresponse to receiving an answer via speech, the device(s) may againutilize the techniques described above to determine whether or not thefirst user actually provided the answer and, hence, whether to complywith the user's answer.

The devices and techniques introduced above may be implemented in avariety of different architectures and contexts. One non-limiting andillustrative implementation is described below.

FIG. 1 shows an illustrative voice interaction computing architecture100 set in a home environment 102 that includes a user 104. Thearchitecture 100 also includes an electronic voice-controlled device 106with which the user 104 may interact. In the illustrated implementation,the voice-controlled device 106 is positioned on a table within a roomof the home environment 102. In other implementations, it may be placedor mounted in any number of locations (e.g., ceiling, wall, in a lamp,beneath a table, under a chair, etc.). Further, more than one device 106may be positioned in a single room, or one device may be used toaccommodate user interactions from more than one room.

Generally, the voice-controlled device 106 has at least one microphoneand at least one speaker to facilitate audio interactions with the user104 and/or other users. In some instances, the voice-controlled device106 is implemented without a haptic input component (e.g., keyboard,keypad, touch screen, joystick, control buttons, etc.) or a display. Incertain implementations, a limited set of one or more haptic inputcomponents may be employed (e.g., a dedicated button to initiate aconfiguration, power on/off, etc.). Nonetheless, the primary andpotentially only mode of user interaction with the electronic device 106may be through voice input and audible output. One exampleimplementation of the voice-controlled device 106 is provided below inmore detail with reference to FIG. 6.

The microphone of the voice-controlled device 106 detects audio from theenvironment 102, such as sounds uttered from the user 104. Asillustrated, the voice-controlled device 106 includes a processor 108and memory 110, which stores or otherwise has access to aspeech-recognition engine 112. As used herein, a processor may includemultiple processors and/or a processor having multiple cores. Thespeech-recognition engine 112 performs speech recognition on audiosignals generated based on sound captured by the microphone, such asutterances spoken by the user 104. The voice-controlled device 106 mayperform certain actions in response to recognizing different speech fromthe user 104. The user may speak predefined commands (e.g., “Awake”;“Sleep”), or may use a more casual conversation style when interactingwith the device 106 (e.g., “I′d like to go to a movie. Please tell mewhat's playing at the local cinema.”).

In some instances, the voice-controlled device 106 may operate inconjunction with or may otherwise utilize computing resources 114 thatare remote from the environment 102. For instance, the voice-controlleddevice 106 may couple to the remote computing resources 114 over anetwork 116. As illustrated, the remote computing resources 114 may beimplemented as one or more servers 118(1), 118(2), . . . , 118(P) andmay, in some instances form a portion of a network-accessible computingplatform implemented as a computing infrastructure of processors,storage, software, data access, and so forth that is maintained andaccessible via a network such as the Internet. The remote computingresources 114 do not require end-user knowledge of the physical locationand configuration of the system that delivers the services. Commonexpressions associated for these remote computing devices 114 include“on-demand computing”, “software as a service (SaaS)”, “platformcomputing”, “network-accessible platform”, “cloud services”, “datacenters”, and so forth.

The servers 118(1)-(P) include a processor 120 and memory 122, which maystore or otherwise have access to some or all of the componentsdescribed with reference to the memory 110 of the voice-controlleddevice 106. For instance, the memory 122 may have access to and utilizea speech-recognition engine 124 for receiving audio signals from thedevice 106, recognizing speech and, potentially, causing performance ofan action in response. In some examples, the voice-controlled device 106may upload audio data to the remote computing resources 114 forprocessing, given that the remote computing resources 114 may have acomputational capacity that far exceeds the computational capacity ofthe voice-controlled device 106. Therefore, the voice-controlled device106 may utilize the speech-recognition engine 124 at the remotecomputing resources 114 for performing relatively complex analysis onaudio captured from the environment 102.

Regardless of whether the speech recognition occurs locally or remotelyfrom the environment 102, the voice-controlled device 106 may receivevocal input from the user 104 and the device 106 and/or the resources114 may perform speech recognition to interpret a user's operationalrequest or command. The requests may be for essentially any type ofoperation, such as database inquires, requesting and consumingentertainment (e.g., gaming, finding and playing music, movies or othercontent, etc.), personal management (e.g., calendaring, note taking,etc.), online shopping, financial transactions, and so forth. In someinstances, the device 106 also interacts with a client applicationstored on one or more client devices of the user 104. In some instances,the user 104 may also interact with the device 104 through this“companion application”. For instance, the user 104 may utilize agraphical user interface (GUI) of the companion application to makerequests to the device 106 in lieu of voice commands. Additionally oralternatively, the device 106 may communicate with the companionapplication to surface information to the user 104, such as previousvoice commands provided to the device 106 by the user (and how thedevice interpreted these commands), content that is supplementary to avoice command issued by the user (e.g., cover art for a song playing onthe device 106 as requested by the user 104), and the like. In addition,in some instances the device 106 may send an authorization request to acompanion application in response to receiving a voice command, suchthat the device 106 does not comply with the voice command untilreceiving permission in the form of a user response received via thecompanion application.

The voice-controlled device 106 may communicatively couple to thenetwork 116 via wired technologies (e.g., wires, USB, fiber optic cable,etc.), wireless technologies (e.g., WiFi, RF, cellular, satellite,Bluetooth, etc.), or other connection technologies. The network 116 isrepresentative of any type of communication network, including dataand/or voice network, and may be implemented using wired infrastructure(e.g., cable, CAT5, fiber optic cable, etc.), a wireless infrastructure(e.g., WiFi, RF, cellular, microwave, satellite, Bluetooth, etc.),and/or other connection technologies.

As illustrated, the memory 110 of the voice-controlled device 106 alsostores or otherwise has access to the speech-recognition engine 112 andone or more applications 126. The applications may comprise an array ofapplications, such as an application to allow the user 104 to make andreceive telephone calls at the device 106, a media player configured tooutput audio in the environment via a speaker of the device 106, or thelike. In some instances, the device 106 utilizes applications storedremotely from the environment 102 (e.g., web-based applications).

The memory 122 of the remote computing resources 114, meanwhile, maystore a response engine 128 in addition to the speech-recognition engine124. The response engine 128 may determine how to respond to voicecommands uttered by users within the environment 102, as identified bythe speech-recognition engine 124 (or the speech-recognition engine112). In some instances, the response engine 128 may reference one ormore user profiles 130 to determine whether and how to respond to speechthat includes a potential valid voice command, as discussed in furtherdetail below.

In the illustrated example, the user 104 issues the following voicecommand 132: “Wake up . . . . Please play my Beatles station”. In thisexample, the speech-recognition engine 112 stored locally on the device106 is configured to determine when a user within the environment uttersa predefined utterance, which in this example is the phrase “wake up”.In response to identifying this phrase, the device 106 may beginproviding (e.g., streaming) generated audio signals to the remotecomputing resources to allow the speech-recognition engine 124 toidentify valid voice commands uttered in the environment 102. As such,after identifying the phrase “wake up” spoken by the user 104, thedevice may provide the subsequently generated audio signals to theremote computing resources 114 over the network 116

In response to receiving the audio signals, the speech-recognitionengine 124 may identify the voice command to “play” the user's “Beatlesstation”. In some instances, the response engine 128 may perform speechidentification or other user-identification techniques to identify theuser 104 to allow the engine 128 to identify the appropriate station. Todo so, the response engine 128 may reference the user profile database130. As illustrated, each user profile may be associated with aparticular voice signature 134 and one or more characteristics 136 inaddition to the voice signature.

For instance, if the response engine 128 attempts to identify the user,the engine 128 may compare the audio to the user profile(s) 130, each ofwhich is associated with a respective user. Each user profile may storean indication of the voice signature 134 associated with the respectiveuser based on previous voice interactions between the respective userand the voice-controlled device 106, other voice-controlled devices,other voice-enabled devices or applications, or the respective user andservices accessible to the device (e.g., third-party websites, etc.). Inaddition, each of the profiles 130 may indicate one or more othercharacteristics 136 learned from previous interactions between therespective user and the voice-controlled device 106, othervoice-controlled devices, or other voice-enabled devices orapplications. For instance, these characteristics may include:

-   -   commands often or previously issued by the respective user;    -   command sequences often or previously issued by the respective        user;    -   grammar typically used by the respective user (i.e., common        phrases used by a user, common patterns of words spoken by a        user, etc.);    -   a vocabulary typically used by the respective user;    -   a language spoken by the respective user;    -   a pronunciation of certain words spoken by the respective user;    -   content to which the respective user has access and/or content        that the respective user often requests;    -   a schedule associated with the respective user, either learned        over time or determined with reference to a calendaring        application associated with the respective user;    -   third-party services that the respective user has registered        with (e.g., music services, shopping services, email account        services, etc.);    -   days on which the respective user often issues voice commands or        is otherwise present in the environment;    -   times of day at which the respective user often issues voice        commands or is otherwise present in the environment;    -   a location of the respective user when the voice-controlled        device 106 captures the audio (e.g., obtained via a GPS location        of a client device associated with the user);    -   previous interactions between the respective user and the        voice-controlled device 106, other voice-controlled devices, or        other voice-enabled devices or applications;    -   background noise that commonly exists when the respective user        interacts with the voice-controlled device 106 (e.g., certain        audio files, videos, television shows, cooking sounds, etc.); or    -   devices frequently detected (e.g., via WiFi) in the presence of        the respective user.

Of course, while a few examples have been listed, it is to beappreciated that the techniques may utilize multiple other similar ordifferent characteristics when attempting to identify the user 104 thatutters a command. For instance, the response engine 128 may referencewhich users have recently interacted with the device 106 in determiningwhich user is likely currently interacting with the device. The amountof influence this factor has in determining which user is interactingwith the device 106 may decay over time. For instance, if one minute agoa particular user made a request to the device, then the device mayweight this interaction more greatly than if the interaction was tenminutes prior. Furthermore, in some instances, multiple user profilesmay correspond to a single user. Over time, the response engine 128 maymap each of the multiple profiles to the single user, as the device 106continues to interact with the particular user.

After identifying the user 104, the response engine 128 may, in thisexample, begin providing the requested audio (the user's Beatlesstation) to the device 106, as represented at 138. The engine 128 mayobtain this audio locally or remotely (e.g., from an audio or musicservice). Thereafter, the user 104 and/or other users may providesubsequent commands to the voice-controlled device. In some instances,only the user 104 and/or a certain subset of other users may providevoice commands that are interpreted by the device to represent validvoice commands that the device 106 will act upon. In some instances, theuser(s) that may provide voice commands to which the device 106 will actupon may be based on the requested action. For instance, all users maybe authorized to raise or lower volume of music on the device 106, whileonly a subset of users may be authorized to change the station beingplayed. In some instances, an even smaller subset of users may beauthorized to purchase items through the device 106 or alter an ongoingshopping or purchase process using the device 106.

Therefore, as the speech-recognition engine 124 identifies speech fromwithin audio signals received from the device 106, the response engine128 may determine whether the speech comes from the user 104 based on avoice signature and/or one or more other characteristics. In oneexample, the response engine 128 may perform an operation requested bythe speech in response to determining that the user 104 uttered thespeech. If, however, the engine 128 determines that the user 104 did notutter the speech, the engine 128 may refrain from performing the action.

FIG. 1, for instance, illustrates two users in an adjacent room having aconversation. A first user of the two users states the followingquestion at 140: “Mom, can I have some more potatoes?” A second user ofthe two responds, at 142, stating: “Please stop teasing your sister andyou can have more.” The microphone(s) of the device 106 may capture thissound, generate a corresponding audio signal, and upload the audiosignal to the remote computing resources 114. In response, thespeech-recognition engine 124 may identify the word “stop’ from theaudio signal. Before stopping the playback of the audio at the device106, however, the response engine 128 may determine whether the userthat stated the word “stop” is the same as the user that issued theinitial command to play the music at the device 106. For instance, theresponse engine 128 may compare a voice signature of the speech at 142to a voice signature associated with the command 132 and/or to a voicesignature of the user 104.

In addition, the response engine 128 may utilize one or morecharacteristics other than the voice signatures to determine whether tointerpret the speech as a valid voice command. For instance, theresponse engine 128 may reference any of the items listed above. Forexample, the response engine 128 may reference, from the profileassociated with the user 104, a grammar usually spoken by the user 104and may compare this to the grammar associated with the speech 142. Ifthe grammar of the speech 142 generally matches the grammar usuallyspoken by the user 104, then response engine 128 may increase thelikelihood that it will perform an operation associated with the command(e.g., will stop playback of the audio). Grammar may include phrasesspoken by a user, common word patterns spoken by a user, words oftenselected by a user from synonyms of the word (e.g., “aint” vs. “isn't”),and the like.

The response engine 128 may also reference the words around thepotential voice command to determine whether the command was indeedintended for the device 106, without regard to whether or not the user104 uttered the speech 142. In this example, for instance, the engine128 may identify, from the words surrounding the word “stop,” that theuser uttering the command was not speaking to the device 106.

Additionally or alternatively, the response engine 128 may compare anyof the characteristics associated with the speech 142 to correspondingcharacteristics associated with the user 104, such as:

-   -   commands often or previously issued by the user 104;    -   command sequences often or previously issued by the user 104;    -   a vocabulary typically used by the user 104;    -   a language typically spoken by the user 104;    -   a pronunciation of certain words spoken by the user 104;    -   content to which the user 104 has access and/or content that the        respective user often requests;    -   a schedule associated with the user 104, either learned over        time or determined with reference to a calendaring application        associated with the user 104;    -   third-party services that the user 104 has registered with        (e.g., music services, shopping services, email account        services, etc.);    -   times of day at which the user 104 often issues voice commands        or is otherwise present in the environment;    -   a location of the user 104 within the environment 102 when the        voice-controlled device 106 captured the speech versus a        location from which the speech 142 originates (e.g., determined        by time-of-flight (ToF) or beamforming techniques);    -   previous interactions between the user 104 and the        voice-controlled device 106, other voice-controlled devices, or        other voice-enabled devices or applications; or    -   devices frequently detected (e.g., via WiFi) in the presence of        the user 104.

Of course, while a few examples have been listed, it is to beappreciated that any other characteristics associated with the speech142 may be used to determine whether the user 104 uttered the speech142.

FIG. 2 depicts a flow diagram of an example process 200 for performing afirst operation in response to receiving a voice command from a user,receiving a second voice command, and performing a second operation inresponse to determining that the user that uttered the first voicecommand also uttered the second voice command. The process 200 (as wellas each process described herein) may be performed in whole or in partby the device 106, the remote computing resources 114, and/or by anyother computing device(s). In addition, each process is illustrated as alogical flow graph, each operation of which represents a sequence ofoperations that can be implemented in hardware, software, or acombination thereof. In the context of software, the operationsrepresent computer-executable instructions stored on one or morecomputer-readable media that, when executed by one or more processors,perform the recited operations. Generally, computer-executableinstructions include routines, programs, objects, components, datastructures, and the like that perform particular functions or implementparticular abstract data types.

The computer-readable media may include non-transitory computer-readablestorage media, which may include hard drives, floppy diskettes, opticaldisks, CD-ROMs, DVDs, read-only memories (ROMs), random access memories(RAMs), EPROMs, EEPROMs, flash memory, magnetic or optical cards,solid-state memory devices, or other types of storage media suitable forstoring electronic instructions. In addition, in some embodiments thecomputer-readable media may include a transitory computer-readablesignal (in compressed or uncompressed form). Examples ofcomputer-readable signals, whether modulated using a carrier or not,include, but are not limited to, signals that a computer system hostingor running a computer program can be configured to access, includingsignals downloaded through the Internet or other networks. Finally, theorder in which the operations are described is not intended to beconstrued as a limitation, and any number of the described operationscan be combined in any order and/or in parallel to implement theprocess.

The process 200 includes, at 202, receiving a first voice commandrequesting performance of a first operation. As described above, thefirst voice command may request performance of any type of operation,such as making a telephone call, playing an audio file, adding an itemto list, or the like. At 204, and in response, the process 200 causesperformance of the first operation. At 206, the process 200 receives asecond voice command requesting performance of a second operation. Inresponse, the process 200 determines, at 208, whether the user thatissued the second voice command is the same as the user that issued thefirst voice command.

In some instances, the process 200 may make this determination withreference to a voice signature comparison 208(1) and a comparison 208(2)of one or more other characteristics. As described above, the voicesignature of the first and second commands may based, respectively, onthe volume, frequency, tone, pitch, or the like of the respectivecommand. In some instances, the process of making this voice signaturecomparison includes first extracting features from the first voicecommand to form the initial voice signature or “voice print”.Thereafter, the second voice command may be compared to the previouslycreated voice print. In some instances, a voice print may be compared toa previously created voice print(s) that occurred in a same session(i.e., one unit of speech may be compared to another unit of speech justuttered). Technologies used to process and store voice prints includefrequency estimation, hidden Markov models, Gaussian mixture models,pattern matching algorithms, neural networks, cepstral mean subtraction(CMS), cepstral variance normalization (CVN), random forest classifiers,matrix representation, Vector Quantization, decision trees, cohortmodels, and world models. In some instances, a voice print may bederived by a joint-factor analysis (JFA) technique, an I-vector approach(based on a JFA), from a cMLLR, from a vocal tract length normalizationwarping factor, or the like.

In some instances, certain factors associated with a user's utteranceare used to determine which speech features to focus on when attemptingto identify a user based on an utterance of the user. These features mayinclude a length of a user's utterance, a signal-to-noise (SNR) ratio ofthe utterance, a desired tradeoff between precision and robustness, andthe like. For instance, a warping factor associated with the userutterance may be used more heavily to perform identification when auser's utterance is fairly short, whereas a cMLLR matrix may be utilizedfor longer utterances.

The characteristic comparison 208(2), meanwhile, may include comparing agrammar of the first command (or of an identified user associated withthe first command) to a grammar of the second command, a location in anenvironment from which the first command was uttered to a locationwithin the environment from which the second command was uttered, and/orthe like. This characteristic may, therefore, include determine howsimilar a grammar of the first command is to a grammar of the secondcommand (e.g., expressed in a percentage based on a number of commonwords, a number of words in the same order, etc.).

If the process 200 determines that the same user issued both the firstand second commands, then at 210 the process 200 causes performance ofthe second operation. In some instances, the process 200 makes thisdetermination if the likelihood (e.g., based on the voice-signaturecomparison 208(1) and the characteristic comparison 208(2)) is greaterthan a certain threshold. If, however, the process 200 determines thatthe same user did not issue the first and second commands, then theprocess 200 may refrain from causing performance of the operation at212. This may further include taking one or more actions, such asquerying, at 212(1), a user within the environment as to whether theuser would indeed like to perform the second operation. If a userprovides an affirmative answer, the process 200 may again determinewhether the user that issued the answer is the same as the user thatuttered the first voice command and, if so, may perform the operation.If not, however, then the process 200 may again refrain from causingperformance of the operation. In another example, the process 200 mayissue the query or request to authorize the performance of the secondvoice command to a device or application associated with the user thatissued the first voice command. For instance, the process 200 may issuesuch a query or request to the “companion application” of the userdescribed above (which may execute on a tablet computing device of theuser, a phone of the user, or the like).

FIG. 3 depicts a flow diagram of an example process 300 for receivingspeech while performing an operation for a user and determining whetherto perform another operation specified by the speech based on aconfidence level regarding whether or not the user uttered the speech.At 302, the process 300 receives, while performing an operation for auser, an audio signal that includes speech. At 304, the process 300identifies the speech from within the audio signal. At 306, the process300 calculates a confidence level that the user uttered the speech. Asillustrated, this may include comparing, at 306(1), a voice signatureassociated with the user to a voice signature of the speech andcomparing, at 306(2), a characteristic associated with the user to acharacteristic associated with the speech. At 308, the process 300 maydetermine whether or not to perform an operation specified by the speechbased at least in part on the confidence level. For instance, theprocess 300 may perform the operation if the confidence level is greaterthan a threshold and otherwise may refrain from performing theoperation. As used herein, a confidence level denotes any metric forrepresenting a likelihood that a particular user uttered a particularpiece of speech. This may be represented as a percentage, a percentile,a raw number, a binary decision, or in any other manner.

FIG. 4 depicts a flow diagram of an example process 400 for determiningwhether or not to interpret speech as a valid voice command based onwhether a user that utters the speech is the same as a user that uttereda prior voice command. At 402, the process 400 identifies a voicecommand uttered by a user. At 404, the process 400 identifies speechthat is subsequent to the voice command. At 406, the process 400attempts to determine whether the user that uttered the voice commandalso uttered the subsequent speech. At 408, the process 408 makes adetermination based on 406. If the process 400 determines that the userdid in fact utter the speech, then the process 400 may interpret thespeech as a valid voice command at 410. If the process 400 determinesthat the user did not utter the speech, however, then at 412 the process400 may refrain from interpreting the speech as a valid voice command.

In some instances, the process 400 may additionally identify the userthat uttered the subsequent speech and may attempt to communicate withthis user to determine whether or not to perform an action associatedwith this speech. For instance, the process 400 may output audiodirected to the user or may provide a communication to a device orapplication (e.g., a companion application) associated with the userthat uttered the subsequent speech to determine whether or not toperform an operation corresponding to this subsequent speech.

FIG. 5 depicts a flow diagram of an example process 500 for receiving afirst voice command, performing a first action in response, anddetermining whether or not to perform a second action associated with asecond voice command based on a characteristic of the second actionand/or the second voice command.

At 502, the process 500 receives a first voice command uttered by a userwithin an environment. For instance, the user may utter a request thatthe voice-controlled device 106 of FIG. 1 play a particular song ormusic station. At 504, the process 500 performs a first action at leastin partly in response to receiving the first voice command. Forinstance, the process 500 may instruct the device 106 to play therequested song or station. At 506, the process 500 receives a secondvoice command uttered within the environment, the second voice commandrequesting performance of a second action that is related to the firstaction. This action may include, in the instant example, turning up thevolume, changing the music station, purchasing a song currently beingplayed, or the like.

At 508, the process 500 identifies one or more characteristicsassociated with the second voice command, the second action, or bothand, at 510, the process determines whether or not to perform the secondaction based at least in part on the characteristic(s). For instance,the process 500 may determine whether the user that uttered the secondvoice command is the same as the user that uttered the first voicecommand, and may perform the action if so, while refraining fromperforming the action if not. In another example, the process 500 mayidentify the user that uttered the second command and determine whetherthis user is one of a group of one or more users authorized to causeperformance of the second action. For instance, certain members of afamily may be allowed to purchase music via the voice-controlled device,while others may not be. In another example, the process 500 may simplydetermine whether the user that uttered the second voice command isknown or recognized by the system—that is, whether the user can beidentified. If so, then the process 500 may cause performance of thesecond action, while refraining from doing so if the user is notrecognized. In another example, the characteristic may simply beassociated with the action itself. For instance, if the second action isturning down the volume on the device 106, then the process 500 mayperform the action regardless of the identity of the user that issuesthe second command.

FIG. 6 shows selected functional components of one implementation of thevoice-controlled device 106 in more detail. Generally, thevoice-controlled device 106 may be implemented as a standalone devicethat is relatively simple in terms of functional capabilities withlimited input/output components, memory and processing capabilities. Forinstance, the voice-controlled device 106 does not have a keyboard,keypad, or other form of mechanical input in some implementations, nordoes it have a display or touch screen to facilitate visual presentationand user touch input. Instead, the device 106 may be implemented withthe ability to receive and output audio, a network interface (wirelessor wire-based), power, and limited processing/memory capabilities.

In the illustrated implementation, the voice-controlled device 106includes the processor 108 and memory 110. The memory 110 may includecomputer-readable storage media (“CRSM”), which may be any availablephysical media accessible by the processor 108 to execute instructionsstored on the memory. In one basic implementation, CRSM may includerandom access memory (“RAM”) and Flash memory. In other implementations,CRSM may include, but is not limited to, read-only memory (“ROM”),electrically erasable programmable read-only memory (“EEPROM”), or anyother medium which can be used to store the desired information andwhich can be accessed by the processor 108.

The voice-controlled device 106 includes a microphone unit thatcomprises one or more microphones 602 to receive audio input, such asuser voice input. The device 106 also includes a speaker unit thatincludes one or more speakers 604 to output audio sounds. One or morecodecs 606 are coupled to the microphone(s) 602 and the speaker(s) 604to encode and/or decode the audio signals. The codec may convert audiodata between analog and digital formats. A user may interact with thedevice 106 by speaking to it, and the microphone(s) 602 captures soundand generates an audio signal that includes the user speech. Thecodec(s) 606 encodes the user speech and transfers that audio data toother components. The device 106 can communicate back to the user byemitting audible statements through the speaker(s) 604. In this manner,the user interacts with the voice-controlled device simply throughspeech, without use of a keyboard or display common to other types ofdevices.

In the illustrated example, the voice-controlled device 106 includes oneor more wireless interfaces 608 coupled to one or more antennas 610 tofacilitate a wireless connection to a network. The wireless interface(s)608 may implement one or more of various wireless technologies, such aswifi, Bluetooth, RF, and so on.

One or more device interfaces 612 (e.g., USB, broadband connection,etc.) may further be provided as part of the device 106 to facilitate awired connection to a network, or a plug-in network device thatcommunicates with other wireless networks. One or more power units 614are further provided to distribute power to the various components onthe device 106.

The voice-controlled device 106 is designed to support audiointeractions with the user, in the form of receiving voice commands(e.g., words, phrase, sentences, etc.) from the user and outputtingaudible feedback to the user. Accordingly, in the illustratedimplementation, there are no or few haptic input devices, such asnavigation buttons, keypads, joysticks, keyboards, touch screens, andthe like. Further there is no display for text or graphical output. Inone implementation, the voice-controlled device 106 may includenon-input control mechanisms, such as basic volume control button(s) forincreasing/decreasing volume, as well as power and reset buttons. Theremay also be one or more simple light elements (e.g., LEDs aroundperimeter of a top portion of the device) to indicate a state such as,for example, when power is on or to indicate when a command is received.But, otherwise, the device 106 does not use or need to use any inputdevices or displays in some instances.

Several modules such as instruction, datastores, and so forth may bestored within the memory 110 and configured to execute on the processor108. An operating system module 616 is configured to manage hardware andservices (e.g., wireless unit, Codec, etc.) within and coupled to thedevice 106 for the benefit of other modules.

In addition, the memory 110 may include the speech-recognition engine112 and the application(s) 126. In some instances, some or all of theseengines, data stores, and components may reside additionally oralternatively at the remote computing resources 114.

Although the subject matter has been described in language specific tostructural features, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thespecific features described. Rather, the specific features are disclosedas illustrative forms of implementing the claims

What is claimed is:
 1. One or more computing devices comprising: one ormore processors; and one or more computer-readable media storingcomputer-executable instructions that, when executed by the one or moreprocessors, cause the one or more processors to perform acts comprising:receiving a first voice command requesting performance of a firstoperation, the first voice command being associated with a first voicesignature; causing performance of the first operation at least partly inresponse to receiving the first voice command; receiving a second voicecommand requesting performance of a second operation, the second voicecommand being associated with a second voice signature; calculating asimilarity between the first voice signature and the second voicesignature; determining, based at least in part on the calculatedsimilarity, whether to: (i) cause performance of the second operation,or (ii) refrain from causing performance of the second operation.
 2. Oneor more computing devices as recited in claim 1, the acts furthercomprising identifying a characteristic, other than the voice signatureof the second voice command, associated with the second voice command,and wherein the determining whether to cause performance of the secondoperation or refrain from causing performance of the second operation isalso based at least in part on comparing the characteristic to acharacteristic associated with the first voice command.
 3. One or morecomputing devices as recited in claim 2, wherein the characteristiccomprises one or more of: a grammar of the second voice command or atime of day at which the second voice command is uttered.
 4. One or morecomputing devices as recited in claim 1, wherein the calculatingcomprises at least one of comparing a volume of the first voice commandto a volume of the second voice command or comparing a frequency of thefirst voice command to a frequency of the second voice command.
 5. Oneor more computing devices as recited in claim 1, the acts furthercomprising issuing a query regarding whether or not to perform thesecond operation after determining to refrain from causing performanceof the second operation.
 6. One or more computing devices as recited inclaim 1, wherein the first and second voice signatures are based atleast in part on at least one of a volume or frequency of the respectivevoice commands.
 7. One or more computer-readable media storingcomputer-executable instructions that, when executed by one or moreprocessors, cause the one or more processors to perform acts comprising:receiving an audio signal generated by a device residing within anenvironment, the audio signal including speech uttered while the deviceperforms a first operation for a user within the environment; performingspeech recognition on the audio signal to identify the speech within theaudio signal; calculating a confidence level that the user uttered thespeech; and determining whether or not to perform a second operation,specified by the speech, based at least in part on the calculatedconfidence level.
 8. One or more computer-readable media as recited inclaim 7, wherein the calculating comprises comparing a voice signatureof the speech to a voice signature associated with the user, wherein thevoice signature of the speech is based at least in part on at least oneof a volume or frequency of the speech and the voice signatureassociated with the user is based at least in part on at least one of avolume or frequency of previous speech uttered by the user.
 9. One ormore computer-readable media as recited in claim 8, the acts furthercomprising identifying a characteristic, other than the voice signatureof the speech, associated with the speech, and wherein the calculatingfurther comprises calculating the confidence level with reference to thecharacteristic.
 10. One or more computer-readable media as recited inclaim 9, wherein calculating the confidence level with reference to thecharacteristic comprises calculating a similarity between thecharacteristic and a corresponding characteristic associated with theuser.
 11. One or more computer-readable media as recited in claim 9,wherein the characteristic comprises a grammar of the speech, and thecalculating includes comparing a similarity between the grammar and agrammar of one or more previous voice commands spoken by the user. 12.One or more computer-readable media as recited in claim 9, wherein thecharacteristic comprises a location within the environment from whichthe speech was uttered, and the calculating includes comparing asimilarity between the location and a location within the environmentfrom which the user uttered a previous voice command.
 13. A methodcomprising: under control of one or more computing devices configuredwith executable instructions, identifying a voice command uttered by auser in an environment; performing speech recognition to identifysubsequent speech uttered within the environment; attempting todetermine whether the user uttered the subsequent speech; andinterpreting the subsequent speech as a valid voice command at leastpartly in response to determining that the first user uttered thesubsequent speech.
 14. A method as recited in claim 13, furthercomprising refraining from interpreting the subsequent speech as a validvoice command at least partly in response to determining that the firstuser did not utter the subsequent speech.
 15. A method as recited inclaim 13, further comprising: identifying a user that uttered thesubsequent speech; and communicating with the user regarding whether ornot to perform an action associated with the subsequent speech.
 16. Amethod as recited in claim 15, wherein the communicating comprisescausing output of audio directed to the user that uttered the subsequentspeech or sending a communication to a device or application associatedwith the user that uttered the subsequent speech.
 17. A method asrecited in claim 13, wherein the attempting comprises comparing a voicesignature of the voice command uttered by the user to a voice signatureof the subsequent speech.
 18. A method as recited in claim 17, whereinthe voice signature of the voice command is based at least in part on avolume or frequency of the voice command and the voice signature of thesubsequent speech is based at least in part on a volume or frequency ofthe subsequent speech.
 19. A method as recited in claim 13, wherein theattempting comprises: comparing a voice signature of the voice commanduttered by the user to a voice signature of the subsequent speech; andidentifying a characteristic associated with the subsequent speech andcomparing the characteristic to a known characteristic associated withthe user.
 20. A method as recited in claim 19, wherein thecharacteristic comprises a grammar of the subsequent speech, a choice ofwords of the subsequent speech, or a location within the environmentfrom which the subsequent speech was uttered.
 21. A method as recited inclaim 13, wherein the attempting comprises comparing past voice commandsuttered by the user to one or more characteristics associated with thesubsequent speech.
 22. A method comprising: under control of one or morecomputing devices configured with executable instructions, receiving afirst voice command uttered by a user in an environment; performing afirst action at least partly in response to receiving the voice command;receiving a second voice command uttered within the environment, thesecond voice command requesting performance of a second action that isrelated to the first action; identifying one or more characteristicsassociated with at least one of the second voice command or the secondaction; and determining whether or not to perform the second actionbased at least in part on the one or more characteristics.
 23. A methodas recited in claim 22, wherein the one or more characteristics comprisewhether a user that uttered the second voice command is the same as theuser that uttered the first voice command.
 24. A method as recited inclaim 22, wherein the one or more characteristics comprise an identityof a user that uttered the second voice command, and the determiningcomprises determining whether the user that uttered the second voicecommand is include in a group of one or more users authorized to causeperformance of the second action.
 25. A method as recited in claim 22,wherein the one or more characteristics comprise whether or not a userthat uttered the second voice command is recognized to the one or morecomputing devices, and the determining is based at least in part onwhether or not the user that uttered the second voice command isrecognized.